Complexity of Constrained VC-Classes

Let F be a class of n-dimensional binary vectors, i.e., functions f : X → {0, 1} where X = [n] ≡ {1, . . . , n} with a VC-dimension V C(F) = d. The classical result of Sauer says that the complexity of F is bounded as |F| ≤ d i=0 n i ≡ S(d, n). How does the complexity decrease as one further constrains the subset of allowed functions in F ? The paper defines a constraining parameter for binary functions, called the margin μh(x, y) of h at x ∈ [n], which is a form of confidence that h takes the value y at x. Let a sample ζ = {(xi, yi)} l i=1, xi ∈ [n], yi ∈ {0, 1}, and for N ≥ 0, consider a class HN (ζ) ⊆ F of functions h having μh(xi, yi) > N , 1 ≤ i ≤ l. The above question is answered by estimating the cardinality |HN (ζ)| as a function of the margin parameter N by E1 = 1+exp(−(l+2(N +1))/n)S(d, n). In the extreme case, where ζ is the maximal-size sample on which every h ∈ HN(ζ) has μζ(h) > N , the estimate is E2 = exp(− exp(−(2N + 1)))(1 + exp(−(l + 2(N + 1))/n))S(d, n)). The latter is exponentially smaller than E1 in the region of N < N ′, where N ′ is approximately (1/2) ln(n).